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ABSTRACT 



Techniques for determining an output rate for a bit stream, 
the output rate being determined by applying information 
read from the bit stream to a model of a receiver for the bit 
stream. The techniques are employed to construct a statis- 
tical multiplexer that multiplexes varying bit-rate MPEG-2 
bit streams onto a satellite up -link. Minimum and maximum 
output rates for each MPEG-2 bit stream are determined 
such that neither a queue for the bit stream in the multiplexer 
nor the bit stream's decoder will underflow or overflow. The 
rates with regard to the decoder are determined using timing 
information read from the bit stream and a general model of 
an MPEG-2 decoder. The multiplexer first allocates each bit 
stream its minimum bandwidth and then allocates any 
remaining bandwidth to the bit streams in proportion to the 
difference between the minimum and maximum output rates 
for the bit streams, with no bit stream receiving more than 
its maximum output rate. If there is not enough bandwidth 
to give every bit stream its minimum rate, bit rates are 
allocated according to priorities assigned to the bit streams. 

23 Claims, 10 Drawing Sheets 
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USING A RECEIVER MODEL TO The MPEG-2 compression scheme represents a sequence 

MULTIPLEX VARIABLE-RATE BIT of video images as a sequence of pictures, each of which 

STREAMS HAVING TIMING CONSTRAINTS must be decoded at a specific time. There are three ways in 

BACKGROUND OF THE INVENTION which pictures may be compressed. One way is intra-coding, 

5 in which the compression is done without reference to any 

1. Field of the Invention other picture. This encoding technique reduces spatial 
The invention has to do with the transmission of variable- redundancy but not time redundancy, and the pictures result- 
rate bit streams, generally and more particularly with the ing from it are generally larger than those in which the 
efficient time multiplexing of several such bit streams onto encoding reduces both spatial redundancy and temporal 
a transmission medium. 10 redundancy. Pictures encoded in this way are called 

2. Description of the Prior Art: FIGS. 1-3 I-pictures. A certain number of I-pictures are required in a 
Anew problem in data transmission is the transmission of sequence, first, because the initial picture of a sequence is 

data that requires a high band width, is bursty, and has necessarily an I-picturc, and second, because I-picturcs 
temporal constraints. Traditionally, data transmission has permit recovery from transmission errors, 
been done on the public switched networks provided by the 15 Time redundancy is reduced by encoding pictures as a set 
telephone companies and on packet networks. The public of changes from earlier or later pictures or both. In MPEG-2, 
switched networks are designed for interactive voice this is done using motion compensated forward and back- 
applications, and so provide relatively low-bandwidth cir- ward predictions. When a picture uses only forward motion 
cuits that satisfy stringent temporal constraints. The packet compensated prediction, it is called a Predictive-coded 
networks are designed for the transfer of data between 20 picture, or P picture. When a picture uses both forward and 
computer systems. The only constraint is that the data backward motion compensated predictions, it is called a 
eventually arrive at its destination. The amount of bandwidth Bidirectional predictive-coded picture, or a B picture in 
available for a transfer depends on the degree of congestion short. P pictures generally have fewer bits than I pictures and 
in the network. The packet networks thus typically make no B pictures have the smallest number of bits. The number of 
guarantees whatever about when or even in what order the 25 bits required to encode a given sequence of pictures in 
data in a burst of data will arrive at its destination. As may MPEG-2 is thus dependent on the distribution of picture 
be seen from the foregoing, neither the telephone network coding types mentioned above, as well as the picture content 
nor the packet network is well-adapted to handle high- itself. As will be apparent from the foregoing discussion, the 
bandwidth bursty data with time constraints. An example of sequence of pictures required to encode the images of the 
such data is digital television which has been compressed 30 "talking heads" will have fewer and smaller I pictures and 
according to the MPEG-2 standard. For details on the smaller B and P pictures than the sequence required for the 
standard, see Background Information on MPEG-land MTV song presentation, and consequently, the MPEG-2 
MPEG-2Television Compression, which could be found in representation of the images of the talking heads will be 
November 1996 at the URL http:/Avww.cdrevolution.com/ much smaller than the MPEG-2 representation of the images 
text/mpeginfo.htm. 35 of the MTV sequence. 

FIG. 1 shows those details of the MPEG-2 standard that The MPEG-2 pictures are being received by a low-cost 

are required for the present discussion. The standard defines consumer electronics device such as a digital television set 

a encoding scheme for compressing digital representations or a set-top box provided by a CATV service provider, The 

of video. The encoding scheme takes advantage of the fact low cost of the device strictly limits the amount of memory 

that video images generally have large amounts of spatial 40 available to store the MPEG-2 pictures. Moreover, the 

and temporal redundancy. There is spatial redundancy pictures are being used to produce moving images. The 

because a given video picture has areas where the entire area MPEG-2 pictures must consequently arrive in the receiver in 

has the same appearance; the larger the areas and the more the right order and with time intervals between them such 

of them there are, the greater amount of spatial redundancy that the next MPEG-2 picture is available when needed and 

in the image. There is temporal redundancy because there is 45 there is room in the memory for the picture which is 

often not much change between a given video image and the currently being sent. In the art, a memory which has run out 

ones that precede and follow it in a sequence. The less the of data is said to have underflowed, while a memory which 

amount of change between two video images, the greater the has received more data than it can hold is said to have 

amount of temporal redundancy. The more spatial redun- overflowed. In the case of underflow, the motion in the TV 

dancy there is in an image and the more temporal redun- 50 picture must stop until the next MPEG-2 picture arrives, and 

dancy there is in the sequence of images to which the image in the case of overflow, the data which did not fit into 

belongs, the fewer the bits that will be needed to represent memory is simply lost. 

the image. FIG. 1 is a representation of a digital picture source 103 

Maximum advantage for the transmission of images and a television 117 that are connected by a channel 114 that 

encoded using the MPEG-2 standard is obtained if the 55 is carrying a MPEG-2 bit stream representation of a 

images can be transmitted at variable bit rates. The bit rates sequence of TV images. In system 101, a digital picture 

can vary because the rate at which a receiving device source 103 generates uncompressed digital representations 

receives images is constant, while the images have varying of images 105, which go to variable bit rate encoder 107. 

number of bits. A large image therefore requires a higher bit Encoder 107 encodes the uncompressed digital representa- 

rate than a small image, and a sequence of MPEG images 60 tionsto produce variable rate bit stream 109. Variable rate bit 

transmitted at variable bit rates is a variable-rate bit stream stream 109 is a sequence of compressed digital pictures 111 

with time constraints. For example, a sequence of images of variable length. As indicated above, when the encoding is 

that shows a "talking head" will have much more spatial and done according to the MPEG-2 standard, the length of a 

temporal redundancy than a sequence of images for a picture depends on the complexity of the image it represents 

commercial or MTV song presentation, and the bit rate for 65 and whether it is an 1 picture, a P picture, or a B picture, 

the images showing the "talking head" will be far lower than Additionally, the length of the picture depends on the 

the bit rate for the images of the MTV song presentation. encoding rate of VBR encoder 107. That rate can be varied. 
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Id general, the more bits used to encode a picture, the better 
the picture quality. 

Bit stream 109 is transferred via a channel 114 to VBR 
decoder 115, which decodes the compressed digital pictures 
111 to produce uncompressed digital pictures 105. These in 5 
turn are provided to television 117. If television 117 is a 
digital television, they will be provided directly; otherwise, 
there will be another element which converts uncompressed 
digital pictures 105 into standard analog television signals 
and then provides those signals to television 117. There may i° 
of course be any number of decoders 115 receiving the 
output of a single encoder 107. 

In FIG. 1, channel 114 transfers bit stream 109 as a 
sequence of packets 113. The compressed digital pictures 
111 thus appear in FIG. 1 as varying-length sequences of 15 
packets 113. Thus, picture Ul(d) has n packets while picture 
111(a) has k packets. Included in each picture 111 is timing 
information 112. Timing information 112 contains two kinds 
of information: clock information and time stamps. Clock 
information is used to synchronize decoder 115 with encoder 20 
107, The time stamps specify when a picture is to be 
decoded and when it is actually to be displayed. The times 
specified in the time stamps are specified in terms of the 
clock information. As indicated above, VBR decoder 115 
contains a relatively small amount of memory for storing 25 
pictures 113 until they are decoded and provided to TV 117. 
This memory is shown at 119 in FIG. 1 and is termed in the 
following the decoder's bit buffer. Bit buffer 119 must be at 
least large enough to hold the largest possible MPEG-2 
picture. Further, channel 114 must provide the pictures 111 30 
to bit buffer 119 in such fashion that decoder 115 can make 
them available at the proper times to TV 117 and that bit 
buffer 119 never overflows or underflows. Bit buffer 119 
underflows if not all of the bits in a picture 111 have arrived 
in bit buffer 119 by the time specified in the picture's time 35 
stamp for decoder 115 to begin decoding the picture HI. 

Providing pictures 111 to VBR decoder 115 in the proper 
order and at the proper times is made more complicated by 
the fact that a number of channels 114 may share a single 40 
very high bandwidth data link. For example, a CATV 
provider may use a satellite link to provide a large number 
of TV programs from a central location to a number of 
CATV network head ends, from which they are transmitted 
via coaxial or fiber optic cable to individual subscribers or 45 
may even use the satellite link to provide the TV programs 
directly to the subscribers. When a number of channels share 
a medium such as a satellite link, the medium is said to be 
multiplexed among the channels. 

FIG. 2 shows such a multiplexed medium. A number of 50 
channels 114(0) through 114(/i) which are carrying packets 
containing bits from variable rate bit stream. 109(0 . . . n) are 
received in multiplexer 203, which processes the packets as 
required to multiplex them onto high bandwidth medium 
207. The packets then go via medium 207 to demultiplexer 55 
209, which separates the packets into the packet streams for 
the individual channels 114(0 . . . n). Asimple way of sharing 
a high bandwidth medium among a number of channels that 
are carrying digital data is to repeatedly give each individual 
channel 114 access to the high bandwidth medium for a short 60 
period of time, termed herein a slot. 

One way of doing this is shown at 210 in FIG. 2. The short 
period of time appears at 210 as a slot 213; during a slot 213, 
a fixed number of packets 113 belonging to a channel 114 
may be output to medium 207. Each channel 114 in turn has 65 
a slot 213, and all of the slots taken together make up a time 
slice 211. When medium 207 is carrying channels like 



4 

channel 114 that have varying bit rates and time constraints, 
slot 213 for each of the channels 114 must output enough 
packets to provide bits at the rate necessary to send the 
largest pictures 111 to channel 114 within channel 114 's 
time, overflow, and underflow constraints. Of course, most 
of the time, a channel's slot 213 will be outputting fewer 
packets than the maximum to medium 207, and sometimes 
may not be carrying any packets at all. Since each slot 213 
represents a fixed portion of medium 207's total bandwidth, 
any time a slot 213 is not full, a part of medium 207 's 
bandwidth is being wasted. 

In order to avoid wasting the bandwidth of medium 207, 
a technique is used which ensures that time slice 211 is 
generally almost full of packets. This technique is termed 
statistical multiplexing. It takes advantage of the fact that at 
a given moment of time, each of the channels in a set of 
channels will be carrying bits at a different bit rate, and the 
bandwidth of medium 207 need only be large enough at that 
moment of time to transmit what the channels are presently 
carrying, not large enough to transmit what all of the 
channels could carry if they were transmitting at the maxi- 
mum rate. The output of the channels is analyzed statisti- 
cally to determine what the actual maximum rate of output 
for the entire set of channels will be and the bandwidth of 
medium 207 is sized to satisfy that actual peak rate. 
Typically, the bandwidth that is determined in this fashion 
will be far less than is required for multiplexing in the 
manner shown at 210 in FIG. 2. As a result, more channels 
can be sent in a given amount of bandwidth. At the level of 
slots, what statistical multiplexing requires is a mechanism 
which in effect permits a channel 114 to have a slot in time 
slice 211 which varies in length to suit the actual needs of 
channel 114 during that time slice 211. Such a time slice 211 
with varying-length slots 215 is shown at 214. 

One method of statistically multiplexing bit streams is 
disclosed in Rao, U.S. Pat. No. 5,506,844, Method for 
Configuring a Statistical Multiplexer to Dynamically Allo- 
cate Communication Channel Bandwidth, issued Apr. 9, 
1996. FIG. 3 is an overview of the method disclosed in the 
Rao application. System 301 includes a set of encoders 
302(0 . . . n) which encode a set of bit streams 105(0 . . . n). 
During a given period of time, termed herein a window, each 
encoder 302(0 encodes at a constant bit rate; however, the 
bit rate may be changed at the beginning of the window. The 
output of an encoder 302(0 is thus a bit stream 108(0 having 
a pieccwise-constant bit rate. The bit streams 108(0 . . . n) 
are input to multiplexer 303, which multiplexes them onto 
medium 207. 

Multiplexer 303 maximizes the use of medium 207 by 
adjusting the bit rates of encoders 302(0 ...«). As 
mentioned above, there is a relationship between bit rate and 
picture quality. Generally, the higher the bit rate, the better 
the picture quality. Consequently, in adjusting the bit rates of 
encoders 302(0 . . . n), multiplexer 303 must be aware of the 
current picture quality of each bit stream and must adjust the 
bit rates not only to maximize the use of medium 207, but 
also to maximize the picture quality of each of the bit 
streams 108(0- 

As Mux 303 operates, it receives information from each 
encoder 107(0 that indicates the picture distortion rate for 
encoder 107(0's current encoding rate (DIF 311(0) and also 
keeps track of the fullness of encoding buffer 307(0 in 
encoder 107(0, as shown by arrow EBF 309(0- Encoding 
buffer 307(0 noIds bit. stream 105(0 while it is being 
encoded, and encoder 107(0 must encode at a rate such that 
encoding, buffer 307(0 neither overflows nor underflows. 
Multiplexer 303 determines from the current distortion rates 
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of the encoders 107 which encoders need to encode at a 
higher bit rate and which can encode at a lower bit rate and 
at the beginning of a window, it adjusts the rate of each 
encoder 107, as indicated by the arrows BRCTL 305(0 . . . 
n), to maximize the picture quality for all of the encoders 
107 while maximizing the degree to which medium 207's 
bandwidth is used. When multiplexer 303 reduces or 
increases an encoder 107(*)'s bit rate, it also reduces or 
increases the size of EBUF 307(0 m uc encoder. 

While the statistical multiplexer of Rao does maximize 
the degree to which medium 207*s bandwidth is used, it has 
a number of shortcomings. Perhaps the most important of 
these is that it adjusts the multiplexing by changing picture 
quality. The system thus cannot guarantee any user a given 
.quality of picture. 

Another shortcoming is that it requires encoders that 
encode digital images as piecewise -constant bit streams. 
Such bit streams have a lower degree of compression than 
variable-rate bit streams; further, the encoding rate and 
therefore the quality of the picture changes at the beginning 
of each window; with sequences of fast changing images, 
this will produce coding artifacts in the pictures. 

Still another is that the multiplexing requires feedback 
from multiplexer 303 to encoders 302(0 . . . n). One 
consequence of this fact is that multiplexer 303 will not 
work with pre-stored sequences of pictures 111; another is 
that in order to use information like encoder buffer fullness 
309 and distortion information 311 to allocate bandwidth in 
medium 207, multiplexer 303 must take into account the 
inner workings of encoder 107. A third is that there must be 
a high-speed connection between multiplexer 303 and each 
encoder 302 to exchange the control information. Finally, 
the bitrate switching of the encoders and the multiplexer is 
difficult to implement, particularly if it is necessary to 
support video inputs having different frame rates. 

It is an object of the invention disclosed herein to over- 
come these shortcomings and thereby to provide an 
improved statistical multiplexer. 

SUMMARY OF THE INVENTION 

The invention is embodied in a multiplexer that over- 
comes the foregoing problems of the prior art by determin- 
ing the rate at which each bit stream is to be output to the 
medium in accordance with rate information obtained by 
applying a model of a receiver for each of the bit streams to 
information contained in the bit stream itself. The rate for 
each bit stream is determined such that the receiver for the 
bit stream will neither underflow nor overflow and the 
multiplexer's use of the medium's bandwidth is maximized. 
Advantages of the multiplexer of the invention include the 
fact that all of the information needed to determine the rates 
is available within the multiplexer itself, the fact that the 
technique will work with any kind of bit stream for which it 
is possible to make a receiver model that is able to determine 
an output rate on the basis of information read from the bit 
stream, and the fact thai adjustment of the output rates does 
not affect the contents of the bit stream. 

In another aspect, the rate information for each bit stream 
is provided as a minimum allowable rate and a maximum 
allowable rate and the multiplexer allocates bandwidth to the 
bit streams by first giving each bit stream its minimum 
allowable rate and then allocating any remaining bandwidth 
to the bit streams up to the maximum rate for each bit 
stream. In the preferred embodiment, allocation of the 
remaining bandwith for a bit stream is done proportionally 
to the difference between the maximum and minimum rates 
for the bit stream. 
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In another aspect of the invention, the multiplexer further 
uses priority information about the bit streams in adjusting 
the rates of the bit streams when there is not enough 
bandwidth for all of the bit streams to receive the minimum 

5 amount. Each bit stream is either a priority one, priority two, 
or priority three bit stream, with priority one bit streams 
being required to meet the time constraints set in the bit 
stream, priority two bit streams being required only not to 
underflow, and priority three bit streams being permitted to 

10 underflow. Where bandwidth is necessary for priority one or 
priority two bit streams, it is taken from priority two and/or 
priority three bit streams. 

Still another aspect of the invention is a multiplexer for 
MPEG-2 bit streams. The statistical multiplexer multiplexes 

15 a plurality of MPEG-2 bit streams and constant bit rate audio 
bit streams onto an output medium. Included in this aspect 
of the invention are a receiver model for MPEG-2 bit 
streams and algorithms for using the model to determine the 
output bit rate for a MPEG-2 bit stream. 

20 These and other aspects and objects of the invention will 
become apparent to those skilled in the arts to which the 
invention pertains upon perusal of the following Detailed 
Description and Drawing, wherein: 

25 BRIEF DESCRIPTION OF THE DRAWING 

FIG. 1 is a diagram showing how digital television 
pictures are encoded, transmitted, and decoded; 

FIG. 2 is a diagram showing multiplexing of variable-rate 
30 bit streams onto a high band width medium; 

FIG. 3 is a block diagram of a prior-art statistical multi- 
plexer; 

FIG. 4 is a block diagram of a statistical multiplexer 
35 which implements a preferred embodiment of the invention; 

FIG. 5 is a more detailed block diagram of a part of the 
statistical multiplexer of FIG. 4; 

FIG. 6 is pseudo-code for the algorithm used to determine 
the bit rate of a channel in the preferred embodiment; 
40 FIG. 7 is a flow chart for the algorithm used to allocate the 
total bit rate of medium 207 among the channels; 

FIG. 8 is a conceptual block diagram of the statistical 
multiplexer; 

4S FIG. 9 is a high-level block diagram of an encoding 
system which includes an implementation of the statistical 
multiplexer; 

FIG. 10 is a more detailed view of the implementation of 
the statistical multiplexer; 
50 FIG. 11 is a detailed view of a channel input block in the 
statistical multiplexer of FIG. 10; and 

FIG. 12 is a flowchart of the minimal bitrate algorithm. 

The reference numbers in the drawings have at least three 
digits. The two rightmost digits are reference numbers 
ss within a figure; the digits to the left of those digits are the 
number of the figure in which the item identified by the 
reference number first appears. For example, an item with 
reference number 203 first appears in FIG. 2. 

60 DETAILED DESCRIPTION 

The following Detailed Description will first present an 
overview of the preferred embodiment, will then provide a 
description of the hardware in which the preferred embodi- 
65 ment is implemented, and will finally provide a detailed 
description of the algorithms used to allocate bandwidth in 
the preferred embodiment. 
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Conceptual Overview: FIG. 8 

FIG. 8 presents a conceptual overview of a statistical 
multiplexer 801 which incorporates the principles of the 
invention. A number n of variable-rate bit streams 109 are 
received in receiver 803, which provides them to bandwidth 
portion controller 805. Bandwidth portion controller 805 
dynamically determines what portion of the bandwidth of 
medium 801 that each bit stream 109(0 is to receive and 
provides a corresponding portion 815(f) of the bit stream to 
transmitter 817, which outputs the portions 815(0 . . . n) it 
receives of each bit stream 109(0 . . . n) onto medium 207. 

Bandwidth portion controller 805 has a number of sub- 
components. There is a transmission controller 807(f) for 
each bit stream 109(f). Each transmission controller 807(0 
contains a bit stream analyzer 809(f) and a receiver model 
811(f). Bit stream analyzer 809(f) collects information from 
bit stream 109(0 an d applies receiver model 811(f) to the 
collected information to determine what rate is required by 
the condition of the receiving device. In the case of a 
MPEG-2 bit stream, the receiving device is a decoder 115(f), 
and for such a decoder, the required rate can be determined 
from the time stamps and the sizes of the pictures making up 
bit stream 109(f). Transmission controller 807(f) applies 
receiver model 811(0 t0 information to determine rate 
information 812(f). Bandwidth allocator 813 receives rate 
information 812(0 . . . n) and uses this information to 
allocate the portion of the bandwidth of medium 207 that 
each bit stream 109(0 * s t0 receive. Having done this for 
each bit stream 109(0 . . . n), it provides a bit stream portion 
815(0 tnat corresponds to the allocated bandwidth to trans- 
mitter 817. 

It is worth noting here that all of the information required 
by the above technique for allocating bandwidth can be 
obtained by applying the receiver models 811 to the infor- 
mation received from the bit streams 109 and that informa- 
tion need only be exchanged between bandwidth allocator 
813 and transmission controllers 807. There is no need 
whatever to receive information from or provide information 
to the encoders 107. Put another way, all of the information 
needed to allocate the bandwidth is available within statis- 
tical multiplexer 801 itself. 

It is also worth noting that the technique of using a model 
of a receiver to control the rate at which a bit stream is output 
to a receiver may be applied in other situations. For example, 
a receiver model could be used to control the rate at which 
a MPEG-2 encoder encoded data. 
Overview of a Preferred Embodiment: FIG. 4 

FIG. 4 provides an overview of a statistical multiplexer 
401 for MPEG-2 bit streams which is implemented accord- 
ing to the principles of the invention. The main components 
of multiplexer 401 are packet collection controller 403, a 
transmission controller 407(0 f° r cacn variable-rate bit 
stream 109(0, a packet delivery controller 419, and a 
modulator 423, which receives the output of packet delivery 
controller 419 and outputs it in the proper form for trans- 
mission medium 207. Packet collection controller 403 col- 
lects packets from variable -rate bit streams 109(0 . . . n) and 
distributes the packets that carry a given bit stream 109(0 10 
the bit stream's corresponding transmission controller 407 
(0. In the preferred embodiment, the packets for all of the bit 
streams 109(0 . . . n) are output to bus 402. Each packet 
contains an indication of which bit stream it belongs to, and 
packet collection controller responds to the indication con- 
tained in a packet by routing it to the proper transmission 
controller 407(f). It should be noted here that the packets in 
each bit stream 109(0 arrive in transmission controller 
407(0 m tDe order in which they were sent by encoder 
107(/). 
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Transmission controller 407(0 determines the rate at 
which packets from its corresponding bit stream 109(0 * s 
output to medium 207. The actual rate determination is made 
by transmission rate controller 413, which at a minimum, 
s bases its determination on the following information: 

for at least a current picture 111 in bit stream 109(0, me 
timing information 112 and the size of the current 
picture. 

a Video Buffer Verifier (VBV) model 415(0, whic h is a 

10 model of a hypothetical bit buffer 119(f). 

VBV model 415(0 uses me liming information and picture 
size information to determine a range of rates at which bit 
stream 109(0 must be provided to the decoder's bit buffer 
119(0 if bit buffer U9(0 is to neither overflow nor under- 

15 flow. Transmission rate controller 413(0 provides the rate 
information to packet delivery controller 419, which uses 
the information from all of the transmission controllers 407 
to determine during each time slice how the bandwidth of 
transmission medium 207 should be allocated among the bit 

20 streams 109 during the next time slice. The more packets a 
bit stream 109(t*) needs to output during a time slice, the 
more bandwidth it receives for that rime slice. 

Continuing in more detail, transmission controller 407 
obtains the timing and picture size information by means of 

25 bit stream analyzer 409, which reads bit stream 109(f) as it 
enters transmission controller 407 and recovers the timing 
information 114 and the picture size 411 from bit stream 
109(0- Bit stream analyzer 409 can do so because the 
MPEG-2 standard requires that the beginning of each picture 

30 111 be marked and that the timing information 114 occupy 
predetermined locations in each picture 111. As previously 
explained, timing information 114 for each picture 111 
includes a clock value and a decoding time stamp. Trans- 
mission controller 407(0 later decoder 115(0 use tnc 

35 clock value to synchronize themselves with encoder 107(f). 
The timing information is found in the header of the PES 
packet that encapsulates the compressed video data. The 
information is contained in the PTS and DTS time stamp 
parameters of the PES header. The MPEG-2 standard 

40 requires that a time stamp be sent at least every 700 msec. 
If a -compressed picture is not explicitly sent with a com- 
pressed picture, then the decoding time can be determined 
from parameters in the Sequence and Picture headers. For 
details, see Annex C of ISO/IEC 13818-1. Bit stream 

45 analyzer 409 determines the size of a picture simply by 
counting the bits (or packets) from the beginning of one 
picture to the beginning of the next picture. 

The timing information and the picture size are used in 
VBV model 415(f). VBV model 415(0 requires the timing 

50 information and picture size information for each picture in 
bit stream 109(f) from the time the picture enters multiplexer 
401 until the time the picture is decoded in decoder 115(f). 
DTS buffer 414 must be large enough to sold the timing 
information for all of the pictures required for the model. It 

55 should be noted here that VBV model 415(f) 's behavior is 
defined solely by the semantics of the MPEG-2 standard, not 
by any concrete bit buffer 119(f). Any bit buffer for a 
working MPEG-2 decoder must be able to provide the 
decoder with the complete next picture at the time indicated 

60 by the picture's timing information; that means that the bit 
buffer 119(f) f° r an Y working MPEG-2 decoder must be at 
a minimum large enough for the largest possible MPEG-2 
picture. Given this minimum buffer size, the timing infor- 
mation for the pictures, and the sizes of the individual 

65 pictures, VBV model 415(f) can determine a rate of output 
for bit stream 109(t) which will guarantee for bit buffers 
119(0 °f working MPEG-2 decoder that each picture 
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arrives in the bit buffer 119(0 before the time it is to be 
decoded and that there will be no overflow of bit buffer 
119(0- 

Details of Transmission Controller 407 and Packet Delivery 
Controller 419: FIG. 5 

FIG. 5 shows the details of a preferred embodiment of 
transmission controller 407 and packet delivery controller 
419. The figure shows three of the n transmission 
controllers, namely transmission controllers 407(. . . k), and 
the two major components of packet delivery controller 419, 
namely central bit rate controller 501 and switch 511. 
Beginning with transmission controller 407(0, in addition to 
transmission rate controller 413, analyzer 409, and VBV 
model 415, transmission controller 409 includes statistical 
multiplexer buffer (SMB) 507, a meter 505 for buffer 507, 
and throttle 509. 

SMB 507(0 is a first-in-first-out pipe buffer which holds 
the bits of bit stream 109(0 while they are in transmission 
control 407(0- In the preferred embodiment, SMB 507(0 
receives pictures 111 in bursts that contain all or almost all 
of the bits in the picture, depends on the picture size and 
maximal bit rate specified by the encoder. Such bursts are 
termed herein picture pulses, and the time period represented 
by such a picture pulse is denoted as T p , which is the inverse 
of video frame rate. For example, l p = 1/29.97=33 ms for 
NTSC video coding. As previously stated, packet delivery 
controller 419 provides packets in time slices 211. The 
length of time of one of these slices is denoted herein as T c . 
In a preferred embodiment, T c is 10 ms. 

SMB 507(0 must of course be large enough to be able to 
accept picture pulses of any size during the time it takes to 
read out the largest expected picture pulse. SMB 507(0 
further must be emptied at a rate that ensures that it cannot 
overflow, since that would result in the loss of bits from bit 
stream 109(0- 11 should not underflow, since that would 
result in the insertion of null packets in the bit stream, 
resulting in the waste of a portion of the multiplexed 
medium. Meter 505 monitors the fullness of SMB 507(0 and 
provides information concerning the degree of fullness to 
TRC 413(0- TRC 413 (0 tnen uses this information to vary 
the range of bit rates that it provides to packet delivery 
controller 419 as required to keep SMB 507(0 fr° m over " 
flowing or underflowing. In other embodiments, the degree 
of fullness from meter 505 can also be fed back to encoder 
107(0 and used there to increase or decrease the encoding 
rate. It should be noted here that feeding back the degree of 
fullness to encoder 107(0 does not create any dependencies 
between statistical multiplexer 401 and a given type of 
encoder 107. Throtde 509, finally, is set by TRC 413 on the 
basis of information 418(0 tnat l{ nas received from packet 
delivery controller 419 to indicate the number of packets 113 
that bit stream 109(0 is to provide to medium 207 in time 
slice 211. 

In determining the range, TRC 413 sets the minimum rate 
for a given time slice 211 to the maximum of the rate 
required to keep SMB 507 from overflowing and the rate 
required to keep VBV model 415(0 fr° m underflowing and 
the maximum rate for the time slice to the minimum of the 
rate required to keep SMB 507 from underflowing and the 
rate required to keep VBV model 415(0 fr° m overflowing. 

Continuing with packet delivery controller 419, packet 
delivery controller 419 allocates the packets 113 that can be 
output during the time slice 211 T c to bit streams 109(0 . . , n) 
as required to simultaneously satisfy the ranges of rates and 
priorities provided by TRC 413 for each transmission con- 
troller 407(0 and maximize the number of packets 113 
output during time slice 211. In the preferred embodiment, 
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controller 419 has two components, central bit rate control- 
ler 501, which is a processor that analyzes the information 
received from each of the transmission rate controllers 413 
in order to determine how many packets from each bit 
stream 109(0 are t0 De output in the next time slice 211, and 
switch 511, which takes the number of packets 113 permitted 
by throttle 509(0 for eacn bit stream 109(0 during the time 
slice 211. Switch 511 is implemented so as to deliver packets 
from each throttle 509(0 sucn tnat me packets are evenly 
distributed across time slice 211. Implementing switch 511 
in this way reduces the burstiness of the stream of packets 
109(0 t0 decoder 115(0 ai *d thereby reduces the amount of 
transport packet buffer needed in decoder 115. Such imple- 
mentations of switch 511 are well-known in the art. 

An important advantage of multiplexer 401, or indeed of 
any statistical multiplexer built according to the principles of 
the invention is that the multiplexer can simultaneously 
multiplex both constant-rate and variable-rate bit streams 
onto medium 207. The reason for this is that as far as 
statistical multiplexer 401 is concerned, a constant-rate bit 
stream is simply a degenerate case: it is a varying-rate bit 
stream whose rate never varies. Thus, with a constant-rate 
bit stream, TRC 413(0 always returns the same rate infor- 
mation 417(0 to packet delivery controller 419. 
Hardware Implementation of a Preferred Embodiment: 
FIGS. 9-11 

A presently-preferred embodiment of the invention is 
implemented as a modification of the PowerVu satellite 
up-link system manufactured by Scientific-Atlanta, Inc. 
(PowerVu is a trademark of Scientific-Atlanta). FIG. 9 is a 
high-level block diagram of the PowerVu up-link system as 
modified to implement the invention. System 901 includes a 
set of encoders 911(0 . . .«). Each encoder 911(0 encodes a 
video input 903(0 ^d an audio input 905(0; mc video input 
is encoded at a constant or variable bit rate and the audio 
input is encoded at a constant bit rate. Each encoder 911(0 
has an output 913(0 which carries the encoded video and 
audio. In the PowerVu system as modified, the outputs 
913(0 ... n) go to statistical multiplexer 915, which outputs 
a constant bit-rate stream 917 to a modulator for transmis- 
sion to a communications satellite. At a high level, operation 
of all of the components of system 901 is supervised and 
controlled by control processor 907, which communicates 
with the other components by means of Ethernet protocol 
909 (Ethernet is a registered trademark of Xerox 
Corporation). In the presently-preferred embodiment, statis- 
tical multiplexer 915 is implemented as a separate chassis 
which need only be coupled to the rest of the PowerVu 
system by encoded data inputs 913(0 . . . n\ Ethernet 
protocol 909, and output 917. 

FIG. 10 shows the preferred embodiment of statistical 
multiplexer 915 in more detail. Multiplexer 915 receives its 
inputs of encoded video and audio from optical fibers. Each 
SWIF receiver 1001(0 receives input from a single optical 
fiber and there are receivers 1001(0 . . . n) corresponding to 
encoders 911(0 ...«). Each receiver converts the informa- 
tion from photons to digital electronic form and outputs it 
via PCR MOD 1005(0 to channel input block 1009(0- PCR 
MOD 1005(0 corrects the clock information in the encoded 
video and audio to compensate for any delays in the encod- 
ing process. The synchronization information needed to do 
this is provided by MSYNC lock up 1003. 

Channel Input 1009(0 ^ an implementation of transmis- 
sion controller 407(0- Channel input 1009(0 employs a 
software implementation of VBV model 415 to dynamically 
determine a current rate at which the input from receiver 
1001(0 must De output to multiplexed output stream 917 and 
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provides that rate information to central bit rate controller Counters 1107 and 1123 are also implemented using gate 
1007, which in turn actually allocates a specific rate to arrays and channel controller 1113 is a digital signal pro- 
channel input block 1009(0- Channel input block 1009(/) cessor. Central bitrate controller 1007 is implemented using 
then outputs bits in its bit stream to bus 1011 at that rate. The a microprocessor with a support 1C 
combined outputs of blocks 1009(0 . . . n) then go via 5 Detailed Description of Algorithms Used to Compute the 
multiplexed output 1013, PCR MOD 1016, and SWIF Output Rate for a Bit Stream 109(0 ^om Statistical Mul- 
transmilter 1017 to output 917. PCR MOD 1016 modifies tiplexer 401: FIGS. 6, 7, and 12 

the clock information in the encoded video again to deal As indicated above, the maximum rate R,^ at which a 

with the time spent in channel input block 1009(f) and transmission controller 407(0 may output packets 113 to 

outputs the bit stream to SWIF transmitter 1017, which 10 medium 207 is determined by the need to keep SMB buffer 

converts the bit stream to a photonic representation and 507(0 from underflowing and bit buffer 119(0 fr° m over_ 

outputs it to an optical fiber. Communication processor 1015 flowing. The minimum rate R min is determined by the need 

provides high level control to central bitrate controller 1007 to keep SMB buffer 507(0 from overflowing and bit buffer 

and also serves as the interface to PCC 907, a control U9(0 from underflowing. Bit buffer 119(0 W *U not under- 

console, and a system which broadcasts status information. 15 flow if all packets belonging to the picture currently being 

Communications processor 1015 also receives MPEG-2 sent arrive in bit buffer 119(0 before the time indicated in the 

service information tables from PCC 907 and provides them DTS stamp for the picture. 

to service information table insertion 1018, which inserts There are thus two maximum rates and two minimum 

them into the bit streams. rates that need to be taken into account in determining R max 

A presently-preferred embodiment of a single channel 20 and R mln : 

input block 1009(0 is shown in moire detail in FIG. 11. The R maxl is the maximum rate at which bit buffer 119(0 in 

main components are packet director 1101, which detects anv MPEG-2 decoder that conforms to the standard 

audio packets, video packets, and headers and routes them to w ill not overflow; 

different components of input block 1009(0, storage 1115 R is lhe maximum ra ie at which SMB 507(0 ^ not 

for the headers, storage 1117 for a FIFO (queue) to hold 25 underflow- 

^T Ck ?K fr ° m ^ ^ ^T^imi " ^ pU L bl ° Ck K** is the minimum rate at which bit buffer 119(0 will 

1009(0 until they are output to data bus 1011, and a bypass ™\ underflow . and ' 

FIFO 1119 which holds the constant bit rate audio packets . . . ' , . „_ .„ 

while they are in input block 1009(0- Output from FIFO * the minimum rate at which SMB 507(0 will not 

1117 is controlled by throttle 1032 under control of throttle 30 „ overflow. _ 

counter 1123, which specifies the number of packets to be V ™ Kin " e n determmed from above four maxuna 

output from FIFO 1117 during a given time slot. Output md m,Duna as foUows: 

from FIFO 1127 is controlled by throttle 1129, which is R ^ is the minimum of R^ and R ma ^. 

controlled by throttle counter 1123. Throttle counter 1123 is Rm* is the maximum of R,^ andR,, 

set by channel controller 1113 in response to the rate selected 35 

by central bit rate controller 1007. Throttle counter 1127, 415(0 that models the fullness and emptiness of bit buffer 

which is for a constant-rate bit stream and does not depend U9(0; what is needed to compute R mfw2 and R^^ is a 

on VBV model 415(0, is set directly by central bit rate measure of the fullness and emptiness of SMB buffer 507(0- 

controller 1007. The model for the fullness of bit buffer 119(0 is termed 

Operation of input block 1009(0 * s as would be expected, herein VBV fullness and the model for the emptiness of bit 

Serial bit stream 1001(0 from SWIF receiver 1001(0 is buffer U9 (0 fe termed herein VBV emptiness. The algo- 

modified by PCR 1005(0 and ^ output to packet director rithms for measuring VBV emptiness and SMB buffer 

1101, which detects packets, determines their types, and emptiness and fullness are simple and will be dealt with first; 

outputs them to the various components of channel input the algorithm for measuring VBV fullness is substantially 

block 1009(0- Packet director 1101 further provides a start 45 more complex. 

of picture interrupt 1103 to channel controller 1113 to la the case of SMB 507(0, the measure of SMB 

indicate that a new picture is being received in SMB FIFO emptiness, E Wfr , is the amount of free space remaining in 

1117. Channel controller 1113 responds to interrupt 1103 by SMB 507(0- For a given time slice T c 2U(m), it is defined 

using picture size information obtained from picture counter as follows: 

1107, header information stored in header storage 1115, and 50 SMfi SJZE p , . 

information about the amount of space left in SMB FIFO ~ ' 5M ^ m) * 

1117 in the VBV model 415(0 t0 obtain maximum and where F W5 is the actual SMB fullness measured by the 

minimum rates at which data must be output from SMB Meter 505. Since there is a maximum size for MPEG-2 

FIFO 1117 to avoid overflow or underflow in SMB FIFO pictures, termed herein VBV_SIZE, the way to prevent 

1117 and overflow or underflow in VBV model 415(0- 55 SMB 507(0 from overflowing is to guarantee that there is 

Channel controller 1113 outputs these rates via 1121 to always an empt) space in SMB 507(0 lnat is larger than or 

central bitrate controller 1007, which selects a rate for the equal to VBV_SIZE. If the free space becomes less than 

next time slice on the basis of the information from channel that, the minimum rate with regard to SMB 507(0, R m£rt2 , 

controller 1113, the current output requirements of all of the must be increased in the next time slice T c (m+1) according 

other channel controllers 113, and the total capacity of the 60 to the algorithm below: 

output stream. Central bitrate controller 1007 returns the ' CI , m/1) , 1X _ , , 

selected rate to channel controller 1113, which sets throttle wwx " OJ 

counter 1132 accordingly. Throttle counter 1132 then deter-. R m(wC is computed as follows: 

mines how many bits are actually output by throttle 1125 _ , , 

during the next time slice. 65 mMX2 ^ ^ ' 

As shown in FIG. 11, packet director 1101 is implemented Continuing with the determination of R minl for the next T c 

by means of gate arrays and a dual port RAM memory. from VBV model 415(0, the rate can be found from the 



11/21/2003, EAST version: 1.4.1 



6,0i 

13 

information in VBV model 415(0 concerning the pictures 
111 in SMB 507(0- The rule is simply this: the minimum bit 
rate must be such that the picture currently being output is 
completely output from SMB 507(0 before the time indi- 
cated by its DTS time stamp. One implementation is 

*«Mi("+l)«pic_residual_bite (qy{DTS_V max -t); 

Here, pic_residual_bits is the number of bits of the 
picture 111 remaining in SMB 507(0, q is the index of the 
picture currently being transmitted from SMB 507(f) and 
q+1, q+2, ... are the indexes of the following pictures, 
DTS_V mw is the time stamp with the most recent time in 
VBV model 415(0, and t is the actual time determined by the 
synchronization time value in the bit stream. 

The above algorithm guarantees that all bits belonging to 
the picture 111 which is currently being delivered to bit 
buffer 119(0 Dave Deen delivered before the decoding 
time DTS^^ arrives. This algorithm may leave only one 
coded picture in the decoder's bit buffer for decoding. While 
this picture could be decoded correctly, a high bit rate will 
be necessary to deliver the next picture on time such that all 
the bits belong to the next picture, p+1, will be available for 
decoding at the next decoding time instance. This require- 
ment will result in a high bitrate requirement for next Tc 
period and will introduce congestion in the delivery media 
at the next Tc period. A better algorithm is one that guar- 
antees at least two pictures (or more, as long as VBV model 
415(/) does not indicate an overflow) in bit buffer 119(0, 
such as the following: 

/i« /J ,i(ffH-l)=pic_residual_ bits(g)/(DTS(?-l)-f) 

In this scheme, the minimal bitrate calculation is slightly 
changed by using the second largest value of DTS in bit 
buffer (119(0, DTS (q-1). That is the time stamp for the 
picture 111 preceding the last picture 111 to be sent to bit 
buffer 119(0- This scheme guarantees that the picture p has 
already be delivered to decoder 115(0 at t=DTS (q-1). Of 
course, it is even better to set up the minimal bit rate so that 
the number of coded pictures in bit buffer 119(0 * s usually 
more than 2. 

Determining VBV Fullness: FIG. 6 

When there is no need to prevent overflow of SMB 507(0, 
the maximum bitrate of bit stream 109(0 is determined from 
the VBV fullness indicated by VBV model 415(0- 

The greater the VBV fullness indicated by the model, the 
less the maximum bitrate. At the beginning of the operation 
of model 415(0, SMB 507(0 is empty and VBV fullness 
indicates that model 415(0 is empty. As soon as bits appear 
in SMB 507(0, central bitrate controller 501 begins output- 
ting them at a predetermined initial rate, for instance, the 
average rate for such variable -rate bit streams. As bits are 
received in SMB 507(0 an d output to medium 207, the 
picture information in VBV model 415(0 is updated each 
time slice. The newly updated information is used to com- 
pute VBV fullness for the next time slice and the VBV 
fullness is used in turn to determine the maximum bit rate 
Rmaxi at which bits will be output on bit stream 109(0 *° r 
the next time period. The computation is the following: 
*^i(™+iMvBv_size-F^>)/r c 

where F vfcv is the VBV fullness measure provided by VBV 
model 415(0 and m and m4l are the current and next time 
slices T e 211. 

In the preferred embodiment, the computation of F vbv (m) 
is governed by the following considerations: 

The calculation requires a computation of the number of 

pictures 111 are currently contained in VBV model 

415(0. 
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The calculation requires a knowledge of how many bits of 
the picture 111 which is currently being transmitted 
from SMB 507(0 presently remain in SMB 507(0. 
The data items used to compute F^ (m) in the preferred 
5 embodiment include the following: 

a. VBV_SIZE, that is, the maximum size of a MPEG-2 
picture. 

b. The absolute maximum bit rate which packet 
delivery controller 419 can provide to bit stream 109(0. 

c. The current time, t, recovered from the clock time 
information of bit stream 109(0- 

d. Data items for each picture presently in SMB 507(0: 
packet_cnt, the number of packets 113 in the picture, 

15 DTS, the time stamp for the picture, q, the index for 
DTS and packet_cnt for the picture currently leaving 
SMB 507(0, ancl r > tne i Qdex * or tnose va hes for the 
oldest picture for which there is still information in 
model 415(0- 

20 e. Status data items in VBV model 415(0 tnat are updated 
every T c 211: pic_cnt_VBV, the number of pictures 
111 which are presently represented in VBV model 
415(0; pic_residual__bit(q), the number of bits of 
picture 111^ that is currently being transmitted to 
25 decoder 115(0 that remain in SMB 507(0; DTS^^ 
the time stamp with the most recent time stamp value 
that is presently in VBV model 415(0; ^ ^vfcv itself- 
As soon as SMB 507(0 begins receiving bit stream 109(0, 
packet delivery controller 419 sets throttle 509(0 to the 
30 initial rate provided by central bit rate controller 509. As 
packets are read from SMB 507(0 at that rate, transmission 
rate controller 413(0 updates DTS_V^, pic_cnt__VBV, 
F^, and pic_residual_bits (q) as required by the transmis- 
sion of pictures from SMB 507(0 to decoder 115(0 and by 
35 the addition of bits to SMB 507(0. algorithm 601 used 
to do this in a preferred embodiment is shown in FIG. 6. 
Section 603 of algorithm 601 shows how the parameters are 
initialized at the time the first picture arrives in SMB 507(0. 
Execution of loop 604 begins when the first bits of the 
40 picture arrive in SMB 507(0. As shown at 605, the loop is 
executed once every T c 211. At the beginning of each 
execution of loop 604, pic_residual_bits is decremented by 
the number of bits that were sent at the rate R(m) previously 
determined for the current T c 211 by central bitrate control- 
45 ler501. 

At 607, F vbv is computed. There are two cases. In the first 
case, shown at 609, the time stamp DTS for the current 
picture r in VBV model 415(0 indicates a time that is after 
the current time t for bit stream 109(0, so decoding of the 

50 picture r cannot yet have begun. Consequently, the bits that 
were sent during the last T c 211 are simply added to the bits 
that are already in VBV model 415(0 an d F V bv is incre- 
mented bad that amount. If the comparison oft and DTS (r) 
indicates that decoder 115(0 has already begun decoding the 

55 picture r, the second case, shown at 611, is executed. 
pic_cnt__VBV is decremented to indicate that one less 
picture is now represented in VBV model 415(0 anc * ^Vtv is 
adjusted by the difference between the number of bits sent 
to decoder 115(0 in the last T c 211 and the total number of 

60 bits in the picture that is no longer represented in VBV 
model 415(0. ^ er picture r is removed from VBV model, 
415(0, tne index r is incremented by 1. 

Block of code 613 deals with the updating that has to be 
done when a picture q has been completely read from SMB 

65 507(0- When that is the case, pic_residual_bits will have a 
value that is less than or equal to 0. The first updating that 
has to be done is shown at 615. The time stamp DTS for the 
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picture 111 that was just sent is now the maximum DTS in Minimal Bitrate Allocation Algorithm, FIG. 12 

Bit buffer 119(0, so DTS_V max is updated with DTS (q). A FIG. 12 shows a flowchart 1201 for this algorithm. The 

picture q has also been added to the pictures represented in algorithm allocates a minimal bitrate to eachTRC 413(0 and 

VBV model 415(0, so pic_cnt_VBV is incremented returns the number of bits still available to be allocated. The 

accordingly. The second updating is at 617. The new current 5 allocation is ordered by priorities, beginning with PL=1, as 

picture is the next picture in SMB 507(0, so q is updated shown in block 1201. The remainder of the flowchart 

accordingly. Similarly, pic_residual_bits is set to the num- consists of an inner loop 1215, which is executed for each 

ber of bits in the new current picture. TRC 413(0 belonging to a given priority and an outer loop 

Allocating the Total Capacity of Medium 207 Among the 1253 which is executed for each priority. The algorithm 

Channels: FIGS. 7 and 12 10 terminates when any of three conditions occurs: 

FIG. 7 shows a flowchart 701 of the CBC control algo- there is no more bandwidth to allocate; 

rithm i that is used to assign the new bitrate for each VBR fates have beeD aUocatfid t0 aU bit streams 1Q9(0 „ ); 

encoder for the next Tc period. The control algorithm is a , ... 

loop 713 that executes each T„. At the start of the loop, Rmio . aUocahons have been made for all of the pnonties 

and Rmax from each TOC(i) are collected. The total avail- 15 ^4'^,°^ ^ T-J?? ' f I 

able bits per Tc parameter, Be, has already been calculated. ^03, the TRC 413(0 to which bandwidth is currently being 

Be will be only updated when there is a change of channel ^1°* TT^ TTk deten ?»f? by R -*> " 

u j -J.!. n C u i u „ A i„ r»„ „ TRC 413(0- The bandwidth is rounded to complete 188 -bit 

bandwidth, Rc. which only happens rarely. Be is calculated . I v/ 1 . . . , , . A . . ■ , , . 

as packets. In decision block 1205, it is determined whether 

^ there is any bandwidth left. If not, branch 207 is taken, 

Bc*Rc*n terminating loop 1215; if there is, loop 1215 continues to 

decision block 1211, where it is determined whether there 

where Tc is in units of seconds. are more bit streams 109(0 having the current priority. If 

Be is divided among the bit stream 109 in accordance with there are, loop 1215 is repeated; otherwise, as indicated by 

the ranges of rates specified by the TRCs (0 . . . n) and in branch 1213, the program enters a new iteration of outer 

accordance with a set of priorities which indicate which bit loop 1213. In that loop, decision block 1215 first checks 

streams 109 are more important. The priorities are provided whether there is another priority level to be processed; if 

by the operator of processor 907 and are set for each bit there is (branch 227), PL is incremented and a new set of 

stream when the multiplexer is initialized for the bit stream. iterations of inner loop 1215 for that priority begins. If there 

In the preferred embodiment, there are three levels of 3Q is no additional priority level, loop 1233 terminates, as seen 

priority, according to the extent to which timely delivery of at branch 1229. 

the pictures in the bit stream is required: Looking at the termination conditions in more detail, if 

PL-1: Every picture in the bit stream will be delivered, there is no more bandwidth to be allocated, branch 1207 is 

and each of them will be delivered on time. taken. In decision block 1217, it is determined whether there 

PL-2: Some picture will always be delivered on time. For 35 are an y bit streams 109(i) for which a minimal bandwidth 

example, a picture may be repeated to keep bit buffer must ^ bc allocated. If there are none, branch 1219 is 

115(i) from underflowing taken m ^ the remaining bandwidth is returned at 1235. If 

PL-3: No time guarantees. The bit stream could even be *™ m . sti , U streams 109(0, the program takes branch 

interrupted to give the channel to another bit stream. 1221 and enters me panic process 1223 which dealswith the 

PL 1 and 2 are used for real-time video programs. PL 3 is 40 P' oblem a u s rel l uired b ? * e P™ 5 of blt f eam , s 109 (°, c 

, C 4 -, , * , *u«* At u-uu ~ «„i n) and then returns the remaining bandwidth at 1235. 

used for preemptible data, that is, data which has no real- . , . « . • ^ . ^ 

. , • c i_ nP 'u ™ Similarly, branch 1229, taken when all priority levels have 

time requirements. Examples or such data are non-real time J> > ... 1 ... 

video programs or non-time-dependent data such as E-mail. bee ° Pressed, returns the remaining bandwidth at 1235 

PL 3 permits full use of the available bandwidth in situations Continuing with panic process 1223, if a bit stream 109(0 

where the sum of the video data is less than the total vailable 45 c u annot receive the ^inimmn rate U requires, one of two 

bandwidth. The total bandwidth available that T c and the lhui B s ma V occur ' Spending on the bit stream: 

priority for each bit stream 109(0 is provided by input block SMB 507(0 may overflow, causing loss of data. 

707. ITie total bandwidth, the priorities, and the maximums bit buffer 119(0 in decoder 115(0 may underflow, causing 

and minimums for the channels arc employed in block 705 interruption of the display of pictures. 

to allocate a minimal bit rate to each bit stream 109(0- 50 m me first case > either the input to SMB 507(0 must De 

Details on the algorithm used to do this will be given below. decreased or the output from bit SMB 507(0 must be 

Once the minimal bit rates for all bit stream 109(0 . . . n) increased. Generally, the second solution can be employed 

have been allocated, the algorithm subtracts the allocated bit in the short term and the first in the longer term. Beginning 

rates from the total bandwidth to determine whether any with the second solution, the extra bandwidth must be taken 

bandwidth remains (709). If none is left, the allocation is 55 from priority 2 and 3 bit streams, beginning with bit streams 

finished and as shown at 711. 721, and 715, the bandwidth 109(0 w i tn priority 3. These bit streams have no time 

allocated to eachTRC 413(0 iS assigned to it (721) and loop constraints and can be denied any bandwidth at all for as 

713 is repeated for the next T c . If there are bits left (branch long as is necessary. Bandwidth can also be taken from 

717), the residual bits are assigned to the bit streams 109(0 priority 2 bit streams 109(0 that have space in their SMBs 

that can take more bits (719). The algorithm for doing this 60 507(0 by having them output a repeat of a picture until the 

is also explained in more detail below. Once the residual bits panic condition is over or until their SMB 507(0 threatens 

have been assigned, blocks 701, 715, and loop 213 are to overflow. Of course, what the repeat produces at the 

executed as described above. There remains, of course, the receiver is a still picture. Because the repeat picture is totally 

possibility that there is not enough total bandwidth to redundant with regard to the picture it is repeating, it always 

perform the allocation of block 705. This worst-case see- 65 has fewer bits than that picture. 

nario is called Panic mode and will be further discussed Given that the reason for the substitution is to free up 

later. bandwidth, it is desirable to make the repeat picture as small 
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as possible. That is achieved by sending a repeat of a coded 
picture that is not used to predict other pictures. B pictures 
fulfill this criterion, as do P pictures that immediately 
precede an I picture in sequences that do not contain B 
pictures. The substitution technique requires that transmis- 5 
sion controller 113 for a PL 2 bit stream respond to an 
indication of a panic from central bitrate controller 1007 by 
reading header information to determine the type and size of 
the picture being output and when it finds the proper kind of 
picture, following it with repeat pictures until the panic is io 
over. 

Where the problem is underflow of bit buffer 119, if the 
bit stream is a priority 1 bit stream, extra bandwidth must 
again be found and the techniques described above must be 
applied. If bit stream 109(/) is a priority 2 bit stream, the is 
techniques described for priority 1 bit streams may be 
employed, or if that is not possible, the bandwidth required 
for the bit stream may be reduced by outputting a minimal- 
sized repeat picture as described above until the panic 
condition is over or until overflow of SMB 507(t) threatens. 20 

Where the problem is the threatened overflow of one or 
more SMB buffers 507, it may also be addressed by decreas- 
ing the bit rate at which the encoders 107 produce data. If the 
encoders 107 are co-located with statistical multiplexer 401, 
feedback from multiplexer 401 to the encoders may be used 25 
to do this. With this kind of feedback, there is no require- 
ment that multiplexer 401 understand the inner workings of 
encoders 107. All that the signal to a given encoder 107(f) 
need indicate is that the encoder must reduce its output rate 
by some amount. Which encoders receive the signal can be 30 
determined in many fashions by multiplexer 401. One 
approach is to reduce the bit rate, (and therefore the image 
quality) in channels on the basis of their priority levels; 
another is to reduce the bit rate in all channels equally. 
Typically, taking bandwidth from other bit streams would be 35 
a short-term solution that would be employed until the 
encoding rate could be changed. In the preferred hardware 
embodiment, short-term panic management is done in cen- 
tral bitrate controller 1007, while long-term panic manage- 
ment is done in control processor 907. 40 
Algorithm for Allocating Residual Bits 

When each of the bit streams 109(i) has received its 
minimum bitrate and there is still bandwidth remaining in 
medium 207, this residual bandwidth B, is allocated among 
the bit streams in the preferred embodiment by allocating 45 
each bit stream 109(t) an additional bit rate AR(z') which is 
proportional to the difference between the maximum and 
minimum bit rates computed by TRC 413(f) for the bit 
stream. AR(i) is calculated in the preferred embodiment as 
follows: 50 



A/T(/) = 



*m«('Wmin(0 Be 
£(* n «f0-**in(0)Tc 



In a preferred embodiment, all of the bit rates involved in the 
above computation are rounded to an integer number of 
packets per second. 

CONCLUSION 60 

The foregoing Detailed Description has disclosed to those 
skilled in the arts to which the invention pertains how to 
make and use apparatus for outputting a variable-rate bit 
stream which uses a model of a receiver of the bit stream and 
information from the bit stream to determine a rate at which 65 
the variable-rate bit stream must be output to avoid overflow 
or underflow in the receiver. The Detailed Description has 



further disclosed a multiplexer which employs the apparatus 
to multiplex a set of variable- rate bit streams onto a 
medium, has disclosed how such apparatus and multiplexers 
may be used with bit streams of video images that are 
encoded according to the MPEG-2 standard, and has given 
algorithms for the use of models of MPEG-2 receivers to 
compute rate requirements. 

The Detailed Description has disclosed the best mode 
presently known to the inventors of implementing their 
apparatus and multiplexers which employ the apparatus; it 
will, however, be immediately apparent to those skilled in 
the arts to which the invention pertains that the invention 
may be employed with variable-rate bit streams other than 
those which are encoded according to the MPEG-2 standard 
and that many other implementations can be made which 
will incorporate the principles of the invention but employ 
different specific implementation techniques. 

For these reasons, the Detailed Description is to be 
regarded as being in all respects exemplary and not 
restrictive, and the breadth of the invention disclosed herein 
is to be determined not from the Detailed Description, but 
rather from the claims as interpreted with the full breadth 
permitted by the patent laws. 

What is claimed is: 

1. An apparatus for controlling a rate at which a bit stream 
encoded at a variable rate is output, the apparatus compris- 
ing: 

a queue for receiving and storing the bit stream; 

a processor for applying a model of a receiver for the bit 
stream to information read from the bit stream to 
determine an output rate for outputting the bit stream; 

a bit stream analyzer for analyzing at least timing infor- 
mation and picture size information in said bit stream; 
and 

output apparatus responsive to the output rate for output- 
ting the bit stream from the queue. 

2. The apparatus set forth in claim 1 wherein: 

the processor determines the output rate such that the 
receiver of the model will neither overflow nor under- 
flow. 

3. The apparatus set forth in claim 2 wherein: 

the processor is further responsive to a state of the queue 
and further determines the output rate such that the 
queue will not overflow. 

4. The apparatus set forth in claim 3 wherein: 

the processor further determines the output rate such that 
the queue will not underflow. 

5. The apparatus set forth in any of claims 1 through 4 
wherein: 

the bit stream is a sequence of components, the compo- 
nents having varying lengths and each component 
including timing information indicating when the 
receiver must process the component and 
the model includes 

a current set of components which is the set of the 
components in the queue together with the set of the 
components that have been sent to but not yet 
processed by the receiver, 
the timing information for the components in the cur- 
rent set, 

the sizes of the components in the current set, and 
a size of a bit buffer in the receiver of the model. 

6. The apparatus set forth in claim 5 wherein: 

the components are digitally-encoded video images. 
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7. The apparatus set forth in claim 6 wherein: 

the video images are encoded according to the MPEG-2 
standard. 

8. The apparatus set forth in claim 2 wherein: 

the bit stream is a sequence of components, the compo- 5 
nents having varying Lengths and each component 
including timing information indicating when the 
receiver must process the component; 
the model includes a0 
a current set of components which is the set of the 

components in the queue together with the set of the 

components that have been sent to but not yet 

processed by the receiver, 
the timing information for the components in the cur- 15 

rent set, 

the sizes of the components in the current set, and 
a size of a bit buffer in the receiver of the model; and 
the processor determines the output rate for a given period 
of time by determining a minimum rate such that each 2 o 
component is output from the queue before the receiver 
must process the component and a maximum rate such 
that the total size of the components in the set of the 
components that have been sent to but not yet pro- 
cessed by the receiver does not exceed the size of the 25 
bit buffer in the receiver of the model. 

9. The apparatus set forth in claim 8 wherein: 

the processor increases the minimum rate for the given 
time period according to the number of bits that remain 
in the queue from a current component that is currently 30 
being output from the queue and the amount of time 
remaining until the current component must have been 
output from the queue and 

the processor decreases the maximum rate for the given 
time period as the total size of the components in the set 35 
of components that have been sent to the receiver and 
not yet processed approaches the size of the bit buffer 
in the receiver of the model. 

10. The apparatus set forth in claim 1, further including a 
bandwidth portion controller coupled to said output appa- 40 
ratus. 

11. The apparatus set forth in claim 10, wherein said 
bandwidth portion controller dynamically determines a vari- 
able portion of said bit stream to be output to a medium. 

12. The apparatus set forth is claim 10, wherein said 45 
bandwidth portion controller is coupled to a plurality of 
output apparatus. 

13. A method of dynamically determining an output rate 
at which a bit stream encoded at a variable rate is output to 

a receiver, the method comprising the steps of: 50 
receiving and storing the bit stream in a queue; 
analyzing said bit stream for at least timing information 

and picture size information; 
in a processor, applying a model of the receiver to 55 

information read from the bit stream to determine the 

output rate; and 
outputting the bit stream from the queue at the output rate. 

14. The method set forth in claim 13 wherein: 

the output rate is determined such that the receiver of the 60 
model will neither overflow nor underflow. 

15. The method set forth in claim 14 wherein the method 
further comprises the step of: 

determining a state of the queue; and 
the step of determining the output rate further determines 
the output rate such that the queue will not overflow. 
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16. The method set forth in claim 15 wherein: 

the step of determining the output rate further determines 
the output rate such that the queue will not underflow. 

17. The method set forth in any of claims 13 through 16 
wherein 

the bit stream is a sequence of components, the compo- 
nents having varying lengths and each component 
including timing information indicating when the 
receiver must process the component, 
the model includes 

a current set of components which is the set of the 
components in the queue together with the set of the 
components that have been sent to but not yet 
processed by the receiver, 
the timing information for the components in the cur- 
rent set, 

the sizes of the components in the current set, and 
a size of a bit buffer in the receiver of the model, and 
the step of determining the output rate is done for a given 
period of time and comprises the steps of 
determining a minimum rate such that the component is 
output from the queue before the receiver must 
process the component and 
determining a maximum rate such that the total size of 
the components in the set of the components that 
have been sent to but not yet processed by the 
receiver does not exceed the size of the bit buffer in 
the receiver of the model. 

18. The method set forth in claim 17 wherein: 

the step of determining the minimum rate increases the 
minimum rate for the given time period according to 
the number of bits that remain in the queue from a 
current component that is currently being output from 
the queue and the amount of time remaining until the 
current component must have been output from the 
queue and 

the step of determining the maximum rate decreases the 
maximum rate for the given time period as the total size 
of the components in the set of components that have 
been sent to the receiver and not yet processed 
approaches the size of the bit buffer in the receiver of 
the model. 

19. The method set forth in claim 17 wherein: 

the components are digitally-encoded video images. 

20. The method set forth in claim 19 wherein: 

the video images are encoded according to the MPEG-2 
standard. 

21. An apparatus for controlling a rate at which an 
encoder encodes a bit stream, the apparatus comprising: 

a queue for receiving and storing the bit stream from the 
encoder; 

a bit stream analyzer for analyzing at least timing infor- 
mation and picture size information in said bit stream; 
and 

a processor coupled to the bit stream and to the encoder 
for applying a model of a receiver for the bit stream to 
the information read from the bit stream to determine 
the rate at which the encoder encodes the bit stream. 

22. The apparatus as set forth in claim 21, further includ- 
ing a bandwidth portion controller. 

23. The apparatus as set forth in claim 22, wherein said 
bandwidth portion controller dynamically determines a vari- 
able portion of said bit stream to be output to a medium. 

***** 
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